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Abstract 

We present a technique for estimating the shape and re¬ 
flectance of an object in terms of its surface normals and 
spatially-varying BRDF We assume that multiple images of 
the object are obtained under fixed view-point and varying 
illumination, i.e, the setting of photometric stereo. Assum¬ 
ing that the BRDF at each pixel lies in the non-negative 
span of a known BRDF dictionary, we derive a per-pixel 
surface normal and BRDF estimation framework that re¬ 
quires neither iterative optimization techniques nor careful 
initialization, both of which are endemic to most state-of- 
the-art techniques. We showcase the performance of our 
technique on a wide range of simulated and real scenes 
where we outperform competing methods. 

Keywords. Photometric stereo, BRDF estimation, Dic¬ 
tionaries, Spatially varying BRDF. 

1. Introduction 

Photometric stereo p7| seeks to estimate the shape of an 
object from images which are obtained from a static cam¬ 
era and under varying lighting. While there has been re¬ 
markable progress in photometric stereo, the vast majority 
of techniques are devoted to scenes that exhibit simple re¬ 
flectance properties. Yet, this creates a significant discon¬ 
nect between theory and practice since the vast majority of 
real-life scenes involve materials with complex reflectance 
properties that interact with light in myriad number of ways. 

In this paper, we present a method for recovering the 
surface normals and the refiectance of opaque objects with 
complex spatially-varying refiectance. The key challenge 
here is that the refiectance, characterized in terms of its 
spatially-varying bidirectional refiectance distribution func¬ 
tion (SV-BRDF), and the shape, characterized in terms of 
surface normals, are coupled and need to be jointly esti¬ 
mated. Further, the SV-BRDF is a very high-dimensional 
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Figure 1. Recovery of surface normals and spatially-varying 

BRDF. We propose a framework for per-pixel estimation of sur¬ 
face normal and BRDF in the setting of photometric stereo. Shown 
above are the estimated shape and rendered images of a visually- 
complex object. The results were obtained from 250 images. 


signal and, in the absence of additional assumptions, re¬ 
quires a large number of input images for robust estimation. 

A common assumption for computational tractability is 
that the BRDF at each pixel is a weighted combination of a 
few reference BRDFs (T4). We now need to estimate only 
the reference BRDFs and their abundances at each pixel 
which is a significant reduction in the dimensionality of 
the unknowns. In Goldman et al. the isotropic Ward 
model, a parametric model is used to characterize the ref¬ 
erence BRDFs. Alldrin et al. assume that the reference 
BRDFs are approximated by the so-called bivariate model, 
a non-parametric model that approximates the 4D BRDF as 
a 2D signal. In both cases, the problem of shape and SV- 
BRDF estimation reduces to alternating minimization over 
the surface normals, the reference BRDFs, and abundances 
of the reference BRDFs at each pixel. The drawback of 
these approaches is that the optimization is not just com¬ 
putationally expensive but also has a critical dependence on 
the ability to find a good initial solution since the underlying 
problem is non-convex and riddled with local minima. 

An alternate approach called example-based photometric 
stereo |[^|^ introduces reference objects — objects with 
known shape — in the scene. These techniques rely on the 




concept of orientation consistency E) which suggests that 
two surface elements with identical normal and BRDF will 
take the same appearance when placed in the same illumi¬ 
nation. If the reference object had the same BRDF as the 
target object, we can obtain the shape of the target simply 
by comparing the intensity profile observed at a pixel on 
the target to those observed on the reference object. When 
the target’s BRDF is spatially-varying, it can be shown that 
two reference objects — one diffuse and the one specular — 
are sufficient to recover the surface normals of the target by 
approximating the unknown BRDF at each pixel as a non¬ 
negative linear combination of the reference BRDFs (n). 
While introducing reference objects is not always desirable, 
example-based photometric stereo produces precise shape 
estimates without requiring the knowledge of lighting. 

The technique proposed in this paper relies on the core 
principle of example-based photometric stereo without ac¬ 
tually introducing reference objects into the scene. Instead, 
given a dictionary whose atoms are BRDFs associated with 
a wide range of materials, we can render virtual spheres, one 
for each atom in the dictionary, under the knowledge of the 
scene illumination (typically a distant point light source). 
This provides a set of “virtual examples” that can be used to 
obtain a per-pixel estimate of the shape and reflectance of 
the scene with arbitrary spatially-varying BRDF (see Fig¬ 
ure]^. The assumption that we make is that the unknown 
BRDF at each pixel lies in the non-negative span of the dic¬ 
tionary atoms. We show that the surface normals and the 
BRDFs can be estimated via a sequence of tractable lin¬ 
ear inverse problems. This obviates the need for complex 
iterative optimization techniques as well as careful initial¬ 
ization required to avoid convergence to local minima. The 
interplay of these ideas leads to a robust surface normal and 
SV-BRDF estimation technique that provides state-of-the- 
art results on challenging real-life scenes (see Figure [^. 

Contributions. We make the following contributions. 

[Model] We propose the use of a dictionary of BRDFs to 
regularize the surface normal and SV-BRDF estimation. 
The BRDF at each pixel of an object is assumed to lie in 
the non-negative span of the dictionary atoms. 

[Normal estimation] We show that the surface normal at 
each pixel can be efficiently estimated using a coarse-to- 
fine search. 

[SV-BRDF estimation] Given the surface normals, we re¬ 
cover the BRDF at each pixel independently by solving a 
linear inverse problem that enforces sparsity in the occur¬ 
rence of the reference BRDFs at the pixel. 

[Validation] We showcase the accuracy of the shape and 
SV-BRDF estimation technique on a wide range of sim¬ 
ulated and real scenes and demonstrate that the proposed 
technique outperforms state-of-the-art. 
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Figure 2. Virtual examples. In example-based photometric stereo 
m , objects with known shape and reflectance are introduced into 
a scene. In contrast, we use a dictionary of BRDFs to render “vir¬ 
tual examples” that guide the normal estimation problem. This 
enables us to handle scenes with very complex reflectance, since 
we can use a larger collection of virtual examples. 


2. Prior work 

In this section, we review some of the key techniques for 
non-Lambertian shape estimation. 

The diffuse + specular model. It is well known that the 
collections of images of a convex Lambertian object typi¬ 
cally lies close to a low-dimensional subspace l[^[T^. This 
naturally leads to techniques p3l[28] - [30| that robustly fit a 
low-dimensional subspace, capturing the Lambertian com¬ 
ponent while isolating non-Lambertian components, such 
as specularities, as sparse outliers. However, these tech¬ 
niques have restrictive assumptions on the range of BRDFs 
to which they are applicable, and more importantly, miss 
out on powerful cues to the shape of the object that is often 
present in specular highlights. 

Parametric BRDFs. Parametric models such as the 
Blinn-Phong 0, Ward pS] , Oren-Nayar fT^ , and Cook- 
Torrance model are based on macro-behavior estab¬ 
lished using specific micro-facet models on the materials, 
and have been widely used in computer graphics. In the 
context of shape and SV-BRDF estimation, Goldman et 
al. utilize the isotropic Ward model | [25| to reduce the di¬ 
mensionality of the inverse problem. Oxholm and Nishino 
EP) further extend this idea by introducing a probabilis¬ 
tic formulation to estimate the BRDFs from a single im¬ 
age under natural lighting conditions. However, parametric 
models are inherently limited in their ability to provide pre¬ 
cise approximations to the true BRDFs and further, lead to 
challenging and ill-conditioned optimization problems. 

Isotropic BRDFs. Isotropic materials exhibit a form of 
symmetry, wherein the reflectance of the material is un¬ 
changed when the incident and outgoing directions are 
jointly rotated about the surface normal. This enables the 








representation of isotropic BRDFs as the function over three 
as opposed to four angles. In the context of photometric 
stereo, Alldrin and Kriegman Q observe that, for isotropic 
materials, the surface normal at each point can be restricted 
to lie on a plane. When the isotropic BRDFs has a single 
dominant lobe, Shi et al. p4| resolve the planar ambiguity 
and show that the surface normals can be uniquely deter¬ 
mined. Higo et al. fT^ utilize properties of isotropy, visi¬ 
bility and monotonicity to restrict the solution space of the 
surface normal at each pixel. This enables a framework for 
shape estimation without the need for radiometric calibra¬ 
tion. Finally, a bivariate approximation for isotropic materi¬ 
als is used in Romeiro et al. |21|22| | to estimate the BRDF of 
a known shape from a single image and without knowledge 
of the scene illumination. 

Reference basis model. As mentioned in the introduc¬ 
tion, a common assumption for scenes with SV-BRDF is 
that the per-pixel BRDF is generated from a few unknown 
reference BRDFs ||^|^[^[^. Invariably, this leads to a 
multi-linear optimization in high-dimensional variables (the 
reference BRDFs) that is highly dependent on initial condi¬ 
tions. In contrast, our proposed technique avoids the need to 
estimate high-dimensional optimization by evoking knowl¬ 
edge of a dictionary of BRDFs. 

3. Problem setup 

Setup. We make the following assumptions. First, the 
camera is orthographic and hence, the viewing direction 
V G is constant across all scene points. Second, the 
scene illumination is assumed to be from a distant point 
light source. The light sources are assumed to be of constant 
brightness (equivalently, that calibration is known) and their 
direction is known. We denote \k G to refer to the light¬ 
ing direction in the k-th image For a light-stage, this 
information is typically obtained by a one-off calibration. 
Third, the effects of long-range illumination such as cast 
shadows and inter-reflections are assumed to be negligible; 
this is satisfled for objects with a convex shape. Finally, the 
radiometric response of the camera is assumed to be known. 

BRDF representation. We follow the isotropic BRDF 
representation used in in which a three-angle coordi¬ 
nate system based on half angles is used. Speciflcally, the 
BRDF is expressed as a function p{0h^ Od^ (t)d) with Oh^Od G 
[0,7r/2) and (j)d G [0,27r). However, by Helmholtz’s 
reciprocity, the BRDF exhibits the following symmetry: 
p{0h, Od, (l)d) = p{0h, Od, (l)d + tt), and hence it is sufficient 
to express (pd ^ [O^tt). Following we use a 1° sam¬ 
pling of each angle. As a consequence, a BRDF is repre¬ 
sented as a point in a T = 90 x 90 x 180 = 1,458, 000- 
dimensional space. When we deal with color images, we 
have a BRDF for each color channel and hence, the dimen¬ 
sionality of the BRDF goes up proportionally. 


Consider a scene element with BRDF p G surface 
normal n, illuminated from a point light source from a di¬ 
rection 1 and viewed from a direction v. For this configu¬ 
ration of normal, incident light and viewing direction, the 
BRDF value is simply a linear functional of the vector p\ 

where S{i v;n} is a vector that encodes the geometry of the 
configuration. In essence, the vector samples the appropri¬ 
ate entry from p, allowing for the appropriate interpolation 
if the required value is off the sampling-grid. 

Problem formulation. Our goal is to recover the surface 
normals and the SV-BRDF in the context of photometric 
stereo; i.e, multiple images of an object ..., ob¬ 
tained from a static camera under varying lighting. The in¬ 
tensity value Ip observed at pixel p = (x^y) with lighting 
\i can be written as 

= (Gii,v;np}Pp) (1) 

where Pp is the BRDF and rip is the surface normal at pixel 
p, respectively, and max{0, rip 1^} accounts for shading. 

Given multiple intensity values at pixel p, one for each 
lighting direction {li,..., Ig}, we can write 





) 




= ^(np)Pp (2) 

Given the intensities. Ip, observed at a pixel p and knowl¬ 
edge of lighting directions {li,..., Ig}, we seek to estimate 
the surface normal rip and the BRDF pp at the pixel. This 
problem is intractable without additional assumptions that 
constrain the BRDF to a lower-dimensional space. 

Model for BRDF. The key assumption that we make is 
that the BRDF at a pixel p lies on the non-negative span of 
the atoms of a BRDF dictionary. Specifically, given dictio¬ 
nary D = [p^, p^, • • • , p^], we assume that the BRDF at 
pixel p can be written as 

Pp — Cp ^ 0, 

where Cp G are the abundances of the dictionary atoms. 

In essence, we have now constrained the BRDF to lie in 
an M-dimensional coneQ This provides immense reduc¬ 
tion in the dimensionality of the unknowns at the expense 
of introducing a model misfit error. Indeed the success of 
this model relies on having a dictionary that is sufficiently 

^ A more appropriate model for the BRDF is that (Dc) > 0. However, 
this leads to significantly higher-dimensional constraints. We instead use a 
sufficient condition to achieve this, c > 0. 
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Figure 3. Accuracy of BRDF models on the MERL database (26) . For the 100 materials in the database, we plot the approximation 
accuracy in relative RMS error (H (also see ([^) for the proposed, bivariate j^, Cook-Torrance ||^, and the Ward p5) models. For the 
proposed model, we use a leave-one-out scheme, wherein for each BRDF the remaining 99 BRDFs in the database are used to form the 
dictionary. The proposed model outperforms competing models both quantitatively (left) as well as in visual perception (right). 


rich to cover a wide range of interesting materials. Figure 
shows the accuracy of various BRDF models on the MERL 
BRDF database 

We also assume that Cp is sparse, suggesting that BRDF 
at each pixel is the linear combination of a few dictionary 
atoms. The sparsity constraint avoids over-fitting to the in¬ 
tensity measurements Ip as well as provides a regularization 
for under-determined problems. 

Solution outline. An estimate of the surface normal and 
BRDF at a pixel p can be obtained by solving 

{np,Cp} = argmin||Ip - A{n)Dc\\l + A||c||i 

(3) 

s.t c>0, ||n ||2 = l 

The -penalty serves to enforce sparse solutions, with 
A > 0 determining the level of sparsity in the solution. 
The optimization problem in ^ is non-convex due to unit- 
norm constraint on the surface normal n as well as the 
term A{n)Dc. Our solution methodology consists of two 
steps: (i) Surface normal estimation. We perform an effi¬ 
cient multi-scale search that provides us with a precise esti¬ 
mate of the surface normal at pixel p (see Section]^; and, 
(ii) BRDF estimation. We solve ^ only over c with the 
normal fixed to obtain the BRDF at p (see Section [^. 

4. Surface normal estimation 

In this section, we describe an efficient per-pixel surface 
normal estimation algorithm. 

4.1. Virtual example-based normal estimation 

Our surface normal estimation is an extension of the 
method proposed in GD, where two spheres — one dif¬ 
fuse and one specular — are introduced in a scene along 
with the target object. To obtain the surface normals at an 
pixel p on the target, the intensity observed at pixel p, Ip, 
is matched to those on the reference spheres. The reference 


spheres provide a sampling of the space of the normals and 
hence, we can simply treat them as a collection of candidate 
normals ff. By orientation consistency, the surface normal 
estimation now reduces to finding the candidate normal that 
can best explain the intensity profile Ip. Given a candidate 
normal n, we have two intensity profiles, 1^(2) and 15(11), 
one each for the diffuse and specular sphere, respectively. 
The estimate of the surface normal at pixel p is given as 

np = argmin min ||Ip - ailD(n) - a2l5(n)||. 
HgA/" ai,a2>0 

In GD, this is solved by scanning over all the pix¬ 
els/candidate normals on the reference spheres. 

Rendering virtual spheres. We rely on the same ap¬ 
proach as GD with the key difference that we virtually ren¬ 
der the reference spheres. The virtual spheres are rendered 
as follows. Given the lighting directions {li,..., 1 q} and 
the BRDF dictionary D = [p^,..., p^], for each candi¬ 
date normal n G A/', we render a matrix B{n) = [bij (n)] G 
X M intensity observed at a surface 

with normal n and BRDF p-^ , under lighting 1^. 

bij{n) = max{ 0 , 

We render one such matrix B{-) for each candidate normal 
in ff. Given these virtually rendered spheres, we can solve 
by searching over all candidate normals. 

Brute-force search. For computationally efficiency, we 
drop the sparsity constraint in We empirically observed 
that dropping the sparsity constraint made little difference 
in the estimated surface normals. Now, given the intensity 
profile Ip at pixel p and noting that B{n) = A{n)D, solv¬ 
ing ^ reduces to: 

Up = argmin min ||Ip — B{n)c\\. (4) 

neAT c>0 

The unit-norm constraint on the surface normals is absorbed 
into the candidate normals being unit-norm. The optimiza¬ 
tion problem in 0 requires solving a set of a non-negative 



























least squares (NNLS) sub-problems, one for each element 
of JV. For the results in the paper, we used the Isnonneg 
function in MATLAB to solve the NNLS sub-problems. 

The accuracy and the computational cost in solving 0 
depends solely on the cardinality of the candidate set AT, 
lA/"! . We obtain JV by uniform or equi-angular sampling on 
the sphere |T0| . As a consequence, the accuracy of the nor¬ 
mal estimates, on an average, cannot be better than the half 
the angular spacing of the candidate set. Yet, the smaller the 
angular spacing of A/', the larger is its cardinality. For ex¬ 
ample, a 5° equi-angular sampling over the hemisphere re¬ 
quires approximately 250 candidates while a 0.5° requires 
20000 candidates. Given that the time-complexity of the 
brute-force search is linear in lA/"!, the computational costs 
for obtaining very precise normal estimates can be over¬ 
whelming. To alleviate this, we outline a coarse-to-fine 
search strategy that is remarkably faster than the brute-force 
approach with little loss in accuracy. 

4.2. Coarse-to-fine search 

Figure]^ shows the value of 

i;(n) = min ||Ip - B(n)c|| 

c>0 

as a function of the candidate normal n for a few exam¬ 
ples. In our simulations, we observed that there is a gradual 
increase in error value as we moved away from the global 
minima of F^(-). We exploit this to design a coarse-to-fine 
search strategy where we first evaluate the candidate nor¬ 
mals at a coarse sampling and subsequently search in the 
vicinity of this solution but at a finer sampling. 

Specifically, let JVq be the set of equi-angular sampling 
on the unit-sphere where the angular spacing is 0 degrees. 
Given a candidate normal n, we define 

Ce{n) = {n | (n,n) > cos 6 », ||n||2 = 1 } 

as the set of unit-norm vectors within 6 >-degrees from n. 

In the first iteration, we initialize the candidate normal 
set = JVe^. Now, at the j-th iteration, we solve (|^ 
over a candidate . Suppose that is the candidate 

normal where the minimum occurs at the j-th iteration. The 
candidate set for the (j + l)-th iteration is constructed as 

j>l, 

with Oj^i < Oj. That is, the candidate set is simply the 
set of all candidates at a finer angular sampling that are no 
greater than the current angular sampling from the current 
estimate. This is repeated till we reach the finest resolu¬ 
tion at which we have candidate normals. For the results 
in this paper, we use the following values: 0i = 10°, O 2 = 
5°, ^3 = 3 °, 6>4 = 1°, and 6>5 = 0.5°. For efficient imple¬ 
mentation, we pre-render 5(n) for n G A/j^ U • • • U A/j^. 


Figure 4. The error as a function of candidate normals for a few 
test examples. We can observe that the global minima is compact 
and the error increases largely monotonically in its vicinity. This 
motivates our coarse-to-fine search strategy. 
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Brute 

time 

0.18s 

0.77s 

4.25s 

27.3s 

74.1s 

force 

ang. error 

7.07° 

3.99° 

1.56° 

0.60° 

0.42° 

Coarse 

time 

0.18s 

0.19s 

0.23s 

0.34s 

0.41s 

to fine 

ang. error 

7.07° 

4.99° 

2.56° 

1.23° 

0.82° 


Table 1. Comparison of brute-force and coarse-to-fine normal es¬ 
timation for different angular samplings in the candidate normals. 
Shown are aggregate statistics over 100 randomly generated trials. 

The computational gains obtained via this coarse-to-fine 
sampling strategy are immense. Table[2 shows the run-time 
and precision of both brute force and coarse-to-fine normal 
estimation strategy for different levels of angular sampling 
in the generation of the candidate normal set. As expected 
the run-time of the brute force algorithm is linear in the 
number of candidates. In contrast, the coarse-to-fine strat¬ 
egy requires a tiny fraction of this time while nearly achiev¬ 
ing the same precision as the brute force strategy. 

While the solution to Q also produces an estimate of the 
BRDF at the pixel, this estimate is often poor due to lack of 
the sparse-regularizer that serves to avoid over-fitting to the 
observed intensities. In the next section, we use this normal 
estimate to obtain a per-pixel BRDF estimate. 

5. Reflectance estimation 

Given the surface normal estimate rip, we obtain an es¬ 
timate of the BRDF at each pixel, individually, by solving 

Cp = argmin ||Ip - B(rip)c ||2 + A||c||i. (5) 

c>0 

The use of the ^i-regularizer promotes sparse solutions and 
primarily helps in avoiding over-fitting to the observed in¬ 
tensities. The optimization problem in ^ is convex and we 
used CVX j^, a general purpose convex solver, to obtain 
solutions. The estimate of the BRDF at pixel p is given as 
Pp = Dcp. The value of A was manually tuned for best per¬ 
formance. For color-imagery, we solve for the coefficients 
associated with each color channel separately. 








When we know a priori that multiple pixels share the 
same BRDF, then we can solve 0 simply by concatenat¬ 
ing their corresponding intensity profiles and their respec¬ 
tive B{') matrices. As is to be expected, pooling intensities 
observed at multiple pixels significantly improves the qual¬ 
ity of the estimates. Yet, while spatial averaging or spatial 
priors improve the quality of the estimate, inherently they 
require the object to exhibit smooth spatial-variations in its 
BRDF. The advantage of our per-pixel BRDF estimation 
framework is the ability to handle arbitrarily complex spa¬ 
tial variations in the BRDF at cost of noisier estimates. In 
the next section, we carefully characterize the performance 
of our proposition using synthetic and real examples. 

6. Results 

We characterize the performance of our technique using 
both synthetic and real datasets. 



Figure 5. Normal and BRDF estimation with varying number 
of images. Given an input number of images, the angular errors (in 
green) and relative BRDF errors (in red) were obtained by averag¬ 
ing across all 100 BRDFs and across 20,000 randomly-generated 
normals per material. 


6.1. Synthetic experiments 

We use the BRDFs in the MERL database | |26| in a leave- 
one-out scheme for testing the accuracy of our proposed al¬ 
gorithms for surface normal and BRDF estimation. Specif¬ 
ically, when we simulate a test object using a particular 
material, the dictionary is comprised of BRDFs of the re¬ 
maining M = 99 materials from the database. We used 
the configuration in the light-stage described in 0 for our 
collection of lighting directions. 

Varying number of images. Figure characterizes the 
errors in surface normal and BRDF estimation for varying 
number of input images or equivalently, lighting directions. 
We report the average error computed by randomly generat¬ 
ing 20,000 normals per material and varying across all 100 
material BRDFs in the database. This experiment is similar 
in setup to the one reported in p4| which, to our knowl¬ 
edge, is one of the most accurate techniques for photomet¬ 
ric stereo on isotropic BRDFs. In p4| , for 200 images, the 
error in estimating the elevation angle when the azimuth is 
known is reported as 0.88°; in contrast, the proposed tech¬ 
nique has an error of 0.82° in estimating the surface normal 
without any prior knowledge of the azimuth. 

Varying BRDF. In Figure we evaluate performance of 
surface normal estimation for varying material BRDFs. We 
fixed the number of images at Q = 253. Shown are aggre¬ 
gate statistics computed over 50,000 randomly generated 
surface normals. The worst case error is less than 2° and 
the error tapers down to 0.5° which is the finest sampling 
that we used for generating candidate normals. This can 
presumably be reduced by either choosing a finer sampling 
grid or using gradient descent techniques. 

Comparisons. Figure [ 7 ] showcases the performance of 
many photometric stereo techniques for different objects: 



Figure 6. Normal estimation for different materials. We fix 

the number of input images/lighting directions to 253. For each 
material BRDF, we compute average error over 50,000 randomly- 
generated surface normals. Inset are the angular error distribution 
for a few select materials. 


a black-obsidian bunny and a gold-painted ele¬ 
phant. We used 253 input images for each object. 
Photometric stereo under Lambertian model fails to re¬ 
cover precise normal maps indicating the presence of non- 
Lambertian components. The robust PCA-based solver 
| [28| produces better normal maps as compared to the tra¬ 
ditional photometric stereo, however it produces overly 
smoothed estimates; this can be attributed to removal of 
non-Lambertian cues which are invaluable for precise nor¬ 
mal estimation. The solution of Alldrin et al. while sig¬ 
nificantly better than Lambertian technique produces errors 
greater than 1°. In contrast, the proposed method returns 
reliable normal estimates for both scenes indicating the ro¬ 
bustness of the underlying solution. We also simulated the 
performance of example-based photometric stereo which is 
identical to the proposed technique when applied to a two- 
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Figure 7. Normal estimation across algorithms. We compare the performance of photometric stereo under Lambertian model (LS), 
robust PCA-based approach (RPCA), simulated example-based fTT] , Alldrin et al. j^l on two objects suing 253 images each. Shown 
are (top-bottom) the estimated surface normals, recovered 3D surface, angular error in normal estimation in degrees and relative error in 
depth map based on different approaches. The insets in rows 3 and 4 are the average normal errors in degrees and the relative depth errors. 



Figure 8. Quantitative BRDF evaluation. Given 253 lighting di¬ 
rections, we evaluate accuracy of BRDF estimation across differ¬ 
ent materials. For each material, we generated 100 normals with 
random orientations and estimated the BRDF for each instance 
individually (per-pixel) as well as collectively. For the per-pixel 
estimates, we plot average and standard deviation of the errors. 

material (white-diffuse and chrome) dictionary. As 
expected, having a larger dictionary of BRDFs as in the pro¬ 
posed technique does provide significant improvements in 
surface normal estimation. 
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Figure 9. Qualitative BRDF evalution. Shown are rendered 
BRDF for the Ceasar statue for a few select material from MERL 
database p6) . (row 1) The rendered image based on ground truth 
BRDF; (rows 2 and 4) rendered images based on estimated BRDF 
from a single normal and 100 randomly generated normals, re¬ 
spectively; (rows 3 and 5) the polar plot for the reflectance func¬ 
tion for the incident light angles [0°, 30°, 60°]. 


BRDF error is given as 


Performance of BRDF estimation. Given a test BRDF, 
we generated 100 surface normals with random orientations 
and rendered their appearance for 253 lighting directions. 
Assuming the knowledge of the true surface normals, we 
estimate the BRDF using the optimization in Section 
We characterize the performance of the per-pixel BRDF 
estimate as well as the error in the BRDF estimate when the 
information at the 100 normals are pooled. We use the rel¬ 
ative BRDF error GD to quantify the accuracy of the esti¬ 
mate. Given true BRDF p and estimated value p, the relative 


'^Wi{{p{i) - p{i)) ■ max(0,cos(6>i)))2/y^u;i, (6) 

i i 

with Wi set equal to 1 for convenience. 

Figure shows the errors for different materials in the 
database — rank-ordered from worst-to-best performance 
— both for the per-pixel BRDF estimation as well as the 
joint estimation. In Figurewe show relighted images and 
polar plots for a subset of materials. 



































Relighted Ground truth Rendering in natural lighting 



Figure 11. Relighting results on “knight_fighting” dataset. 


Figure 10. Results on “gourdl” dataset. We show the estimated 
normal map in false color (top-left) and 3D surface (top-right) re¬ 
covered from it. We also show the relighting results (bottom-left), 
ground truth under the same lighting direction (bottom-middle), 
and relighting under natural environment (bottom-right). 

6.2. Real data 

Real images are present a layer of difficulty well 
beyond simulations and introduce inter-refiections, sub¬ 
surface scattering, cast shadows, and imprecise light source 
localization. We test the performance of our shape and 
BRDF recovery algorithm on a wide range of datasets. 
Specifically, we use images from two sources — the light 
stage data from Q, and the gourd from 0. 

Figures [TT] and [T^ showcase the performance of 
our algorithm on the real datasets. The results in Figure 
[TT]and[T^were obtained from 250 input images, and the re¬ 
sults of “gourd 1” in Figure was obtained from 100 input 

images. The recovered shape and BRDF (as visualized via 
rendered images) seem to be in agreement with the results 
in 0; however, our algorithm is significantly simpler and 
employs a per-pixel algorithm that be easily parallelized. 

The robustness of the per-pixel BRDF estimate is tested 
in Figure where there are not just a wide variety of 
unique materials (the helmet, the breast-plate, the chain, the 
red scabbard, to name a few) but also significant modeling 
deviations (inter-refiections, cast-shadows). In spite of this, 
our approach produces a faithful rendition of the scene. The 
per-pixel BRDF estimation allows us to handle objects with 
complex spatial variations. In contrast, methods that as¬ 
sume the presence of just a few reference BRDFs as in | [2|[8| 
would not scale easily to such scenes. We refer the reader to 
the supplemental videos highlighting the relighting results. 


significantly simplifies the inverse problem and provides not 
just state-of-the-art results in normal and BRDF estimation 
but also works robustly on a wide range of real scenes. A 
hallmark of our approach is the ability to obtain per-pixel 
BRDF estimation without any spatial smoothness assump¬ 
tions endemic to state-of-the-art SV-BRDF estimation tech¬ 
niques 0i; this makes it applicable to scenes with a large 
number of unique materials. Finally, our per-pixel frame¬ 
work is ripe for further speed-ups by solve for the shape 
and refiectance at each pixel in parallel. 

Limitations. While the use of virtual examples provides 
flexibility beyond we require light calibration and 
hence, our method is most suited to shape and reflectance 
acquisition from light-stages where the light sources are 
fixed and the calibration is a one-time effort. The accuracy 
of our coarse-to-fine normal estimation is lower bounded 
by finest sampling of our candidate normals. This can po¬ 
tentially be improved by refine the estimates using a gra¬ 
dient descent scheme starting with our solution; however, 
this approach could be computationally intensive. The SV- 
BRDF produced by our approach can be noisy especially 
since we independently recover the BRDF at each pixel. 
If we have a priori knowledge that the scene has a limited 
number of unique materials, then enforcing this could lead 
to robust SV-BRDF estimates. This can be easily incorpo¬ 
rated into our framework by enforcing the matrix of sparse 
coefficients [ci... Cp ...] to be low-rank. Finally, it is also 
important that the scene lies in the linear span of our dictio¬ 
nary. In the failure of this, our results can be unpredictable. 
Here, the need for a larger dictionary encompassing hun¬ 
dreds, if not thousands, of materials would be invaluable 
for the broader applicability of our method. 


7. Discussions 

We present a photometric stereo technique for per-pixel 
normal and BRDF estimation for objects that are visually 
complex. We demonstrate that the use of a BRDF dictionary 
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